Practical Case Analysis Of Ab Testing And Optimization Closed-loop Process During Japanese Website Group Promotion

1.

overview of project background and goals

(1) target audience: japanese market station group (a total of 12 sub-stations, covering tokyo/osaka traffic).
(2) promotion goal: improve paid conversion rate, reduce bounce rate and ensure stability.
(3) initial traffic: average daily uv≈28,000, peak concurrency≈4,200 rps.
(4) initial performance: ttfb mean ≈ 600ms, lcp ≈ 3.6s, conversion rate cr ≈ 1.8%.
(5) technical constraints: use multiple domain names (including .jp/.com), and some sites use domestic cdn edge nodes.

2.

a/b testing key indicators and sample design

(1) core kpis: cr (conversion rate), bounce (bounce rate), lcp/ttfb, server cpu and bandwidth usage.
(2) sample distribution: traffic is randomly divided 50/50 for 14 days to cover peaks and valleys within the week.
(3) statistical threshold: the minimum detectable improvement is set to 20% (α=0.05, β=0.8).
(4) monitoring items: rps, 5xx error rate, cache hit rate, bandwidth peak.
(5) data collection: combined with ga/server logs/prometheus + grafana aggregation indicators.

3.

server and domain name architecture design (sample configuration)

(1) origin vps (tokyo) example: 8 vcpu, 16gb ram, nvme 500gb, public network 1gbps.
(2) standby node (osaka): 4 vcpu, 8gb ram, nvme 250gb, public network 500mbps.
(3) load balancing: use cloud lb + dns polling, and the health check interval is 10 seconds.
(4) domain name strategy: use .jp for the main domain (seo priority), and .com for the backup domain; dns ttl 60s for switching.
(5) certificate and tls: tls1.3, ocsp stapling, automatic renewal using let's encrypt.

4.

technical implementation of cdn and caching strategies

(1) cdn provider: cloudflare (pro) does edge caching and waf; at the same time, it connects to the local isp cache node.
(2) caching strategy: html page edge caching ttl 120s, static resource caching ttl 7 days.
(3) optimization methods: enable http/2, brotli compression, resource merging and priority loading of key css.
(4) cache hit improvement: from initial 30% to 70% (measured after deployment).
(5) impact assessment: the increase in edge hit rate reduces origin bandwidth by approximately 58% and average response time by approximately 250ms.

5.

ddos defense and high availability configuration

(1) protection solution: cloudflare waf+rate limiting+bgp anycast is linked with the upstream cleaning center.
(2) historical attacks: it has encountered traffic attacks with peak peaks of about 12 gbps/220k rps, and automatically switched to cleaning nodes.
(3) automation strategy: the traffic sudden increase threshold is set to 3 times the baseline to trigger current limiting and black hole protection.
(4) local mitigation: origin network stack tuning (net.core.somaxconn=65535, tcp_tw_reuse=1, etc.) reduces the pressure on half-open connections.
(5) recovery strategy: when a fault occurs, dns ttl=60s is quickly directed to the backup node and triggers an engineering alarm.

6.

a/b test results and data display

(1) comparison dimensions: ttfb, lcp, cr, bounce, average cpu usage.
(2) experimental plan: a=baseline (no edge optimization), b=enable cdn caching+brotli+http/2+caching rules.
(3) operation period: 14 days, sample size a≈120k sessions, b≈118k sessions.
(4) conclusion overview: plan b is significantly better than plan a in key seo and conversion.
(5) the specific data are as follows:

index	a (baseline)	b (after optimization)	improve
ttfb(ms)	600	150	-75%
lcp(s)	3.6	1.8	-50%
conversion rate cr	1.8%	2.6%	+44% relative improvement
bounce rate	62%	48%	-14pp
average cpu usage	55%	30%	-25pp

7.

optimized closed-loop process and configuration examples (implementation suggestions)

(1) closed-loop steps: hypothesis → configuration changes → grayscale/ab testing → monitoring and statistics → decision (release/rollback).
(2) configuration example (nginx): worker_processes auto; keepalive_timeout 65; gzip on; brotli on.
(3) automation: use ci/cd to deploy edge rules and caching policies, and test traffic is controlled through feature flag.
(4) documents and sop: record the domain name switching process, cleaning center linkage contacts and rollback scripts.
(5) long-term iteration: every data change of ≥10% triggers a second experiment to ensure two-way optimization of seo and user experience.